Skip to content

Delegate nnmCorrect() to the NPM package - #521

Open
ivokwee wants to merge 2 commits into
develfrom
use-npmatch-nnmcorrect
Open

ivokwee wants to merge 2 commits into
develfrom
use-npmatch-nnmcorrect

Conversation

@ivokwee

@ivokwee ivokwee commented Sep 8, 2026

Copy link
Copy Markdown
Member

Summary

  • nnmCorrect() (in R/pgx-correct.R) and NPM::NPmatch() (https://github.com/bigomics/NPM) are the same nearest-pair-matching batch correction algorithm, maintained as two separately-drifting implementations. Replace nnmCorrect()'s body with a thin wrapper delegating to NPM::NPmatch(), preserving the original argument names (use.covariates) so existing callers throughout playbase (pgx.superBatchCorrect, runBatchCorrectionMethods, etc.) are unaffected.
  • This pulls in NPM's exact closed-form replacement for the limma::removeBatchEffect() pairing-effect-removal step (mathematically identical, not an approximation), which is much faster because it avoids fitting a regression with one factor level per sample -- the cost that made nnmCorrect slow as sample count and/or number of condition groups grew.
  • Adds NPM to Imports and to Remotes (url::https://github.com/bigomics/NPM/archive/HEAD.zip), following the existing pattern used for other bigomics GitHub packages (playdata, plaid, metaLINCS).
  • nnmCorrect2() is left untouched -- it's a genuinely different, SVD-compressed-covariate approximation, not superseded by this change (see below).

Why

  • nnmCorrect's dominant cost is limma::removeBatchEffect(full.X, batch = full.pairs, design = design), where full.pairs has one factor level per original sample. Because the pairing construction always produces a complete, balanced two-way layout (full.pairs x full.y), the OLS batch coefficient this regression fits collapses to a closed form -- each sample's own mean deviation from the grand mean -- computable with a few vectorised matrix subtractions instead of an n-level factor regression.
  • Benchmarked against nnmCorrect2 (which tackles the same slowness via per-group KNN + truncated-SVD-compressed covariates): the closed-form fix is faster in every case tested and is exact, whereas nnmCorrect2's output reaches only ~0.85 median gene-wise correlation with the exact result (max deviation > 2x the data's SD on GSE10846) -- consistent with it being harder to control/tending to overcorrect in practice.

Speed (output numerically identical, diffs ~1e-12/4e-13 -- floating-point noise, not an approximation)

Real data, GSE10846 (n=350 samples):

knn=1 knn=2 (default)
before 15.4s 31.3s
after 1.7s 1.9s
speedup 9x 16x

Synthetic, scaling with number of condition groups g (n=400 fixed, knn=1):

g before after speedup
2 2.4s 0.5s 5x
4 4.2s 0.5s 9x
8 8.3s 0.5s 16x
12 12.5s 0.6s 22x

Synthetic, scaling with sample count n (g=4 fixed, knn=1):

n before after speedup
200 1.4s 0.25s 6x
400 4.3s 0.5s 9x
800 17.7s 1.4s 12x
1200 44.6s 2.9s 15x

The speedup grows with both n and g -- exactly the two dimensions along which nnmCorrect used to get slow -- because the old cost was dominated by an n-level factor regression that the closed form eliminates.

Test plan

  • Verified the new nnmCorrect() wrapper produces output numerically identical (diffs ~1e-12/4e-13, floating-point noise) to the original inline implementation on GSE10846, at both knn=1 and the default knn=2
  • Run the existing playbase test suite / pgx.superBatchCorrect end-to-end with bc="NPM" on a real project
  • Confirm remotes::install_deps() / CI can resolve the new NPM dependency from the Remotes: URL

🤖 Generated with Claude Code

https://claude.ai/code/session_01KRrp7pzhFakWWzbD9j1i5R

…ementation

playbase's nnmCorrect() and NPM's NPmatch() (https://github.com/bigomics/NPM)
are the same nearest-pair-matching batch correction algorithm maintained in
two places, which had already drifted apart (missing NA-handling, no
small-sample/malformed-pairing guards). Replace nnmCorrect()'s body with a
thin wrapper delegating to NPM::NPmatch(), keeping the original argument
names (use.covariates) so existing playbase callers are unaffected.

This also picks up NPM's closed-form replacement for the
limma::removeBatchEffect() pairing-effect-removal step, which is
mathematically exact but 15-20x faster since it avoids fitting a
regression with one factor level per sample.

Verified on GSE10846 (n=350): output numerically identical to the previous
inline implementation (diff ~4e-13) at both knn=1 and the default knn=2,
running in ~2s instead of ~15-31s.

nnmCorrect2() is left untouched -- it uses a genuinely different,
approximate (SVD-compressed covariate) approach, not superseded by this
change.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KRrp7pzhFakWWzbD9j1i5R
@ivokwee
ivokwee requested a review from zitoa September 8, 2026 16:26
@ESCRI11

ESCRI11 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

@ivokwee should this target devel instead of main?

@ivokwee
ivokwee changed the base branch from main to devel September 9, 2026 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants